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ABSTRACT 

This paper reports on the results of a usability study that 
investigated the suitability of using an open systems approach for developing 
an online assessment application. Issues examined included ease of use, 
security, effectiveness, and ease of maintenance. Participants were 68 
preservice teachers taking a technology integration course. Students were 
administered a Web-based demographics survey, followed by an exam of 25 
multiple choice questions. Results indicate that an open systems approach was 
a feasible way to design instructional tools. Data encapsulation using XML 
(Extensible Markup Language) was easily created, maintained,' and modified, 
allowing for a seamless interaction on the part of the faculty. The students 
reported in exit interviews that taking the exam was easy, and they liked the 
convenience of immediate scores. (Contains 11 references.) (MES) 
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Abstract EducatiOTal technology is at a crossroads. The Internet was origmally designed for the 
needs of tl^ science and mihtary, not for the purposes of sound instroctioiL New and more 
powerfiil protocol schemes are needed to truly realize the potential of networiced-based instructioa 
A new technology that holds a lot of promise is XML. XML provides a more structure and 
dynamic qiproach to content creation that allows the developer to define and order data instead of 
merely di^laying it We were interested in some of the issues in using a new technology such as 
XML for instniction^ plications. In this study we found that the Internet and XML were suitable 
for this type of plication. We found no difference in performance ftom either local accessed 
classes and those delivered ftom an Internet server. 

Introduction 

Edt^ational technology is at a crossroads. R^id development of computer technology bag provided the 
in^vidual ^th an unprecedented access to information and resources that only a generation ago was the stuff of 
science fictioa Never in our history has technology been so important in the success or feilure of our students. 
However, the totemet was originally designed for the needs of the science and military, not for the purposes of 
soxmd mstructioa Consecpiently, newer tools and protocols are needed that truly facilitate teaching and evaluation 
and put mto the hands of teachers the power to design. One area that is critical to student success is student 
assessm oit T his paper reports on the results of a usability study that investigated the suitability of using an open 
systems appr^h for developing an online assessment ^licatioa We were interested in the following issues: ease 
of use, security, effectiveness and ease of maintenance. 

XML and data encapsulation 

The pionps of the Intpet needed a blueprint for a system that relied on a simple message format in 
^njunction with single interactive software. They created a system that would work even though they HaH 
incompatible computer resources. An environment evolved that fostered interoperability and sharing at the 
hardware, software and human levels" (Berger, 1999 p. 27). Secondly, they created standard interaction protocols 
tot w^ "independent of to p^cular transmission subsystem and recjuires only a reliable ordered data stream 
channel. (Po^l, 1992). With this information anyone who can write a computer program, in almost any language, 
can communicate through the Internet as long as to proper interaction protocol is mainteined 

The protocol model greatly increases access because it separates content from the 
delivery tohnology. By ignoring the delivery platform, incompatible systems can communicate 
and share informtion. This approach is superior to the traditional courseware development 
mcxlel because it increases access, improves flexibility and adaptability as well as reducing cost 
through more efficiently maintaining the installed technology base. What is needed is a 
mechamsm for developing the appropriate educational protocols. 

The open protocol model is at a great disadvantage because to classroom has many distinct instructional 
paradigms for content delivery such as the le<toe, class recitation, or small-groiq) collaborative discussions as well 
as to stodent evaluation. Programmers working within the traditional method of courseware development were 
mxwh better able to mana ge the sequencing of lesson content because toy controlled how a specific user interacted 
and navigated through the content. 

A further problem is with HTML, the xmderlying encapsulation language. HTML was 
designed to simply display static document types such as reports or journal articles. It was never 
designed to structure, order, or define data. **HTML formats how you present a Web page's data and it 
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is not designed to work with what that data represents.” (Hohner, 1998, p. xi) Recognizing this discrepancy there 
has been a call for the development of better markup l^guages that could represent not only the format but also the 
structure of the data. This would allow for a much more dynamic and flexible data delivery system. 

The World Wide Web Consortium (see W3C) is developing the Extensible Markiq) Language (XML). XML is 
similar to HTML in that it is a simple text-based system. However, XML provides a more structured and ordered 
approach to content creation. XML is "most often used as data-description language allowing us to organize data 
into data structures - even complex data structures...You can tailor the data as you want it; the most attractive feature 
of XML is that you can create your own tags." (Holzner, 1998", p.4). This lets an author structure data into 
^propriate elements "without having to restructure your data to fit a predefined maikiq) language." (Holzner, 1998, 
p. xi). 

Because the markiq) tags in XML are not fixed ahead of time, as they are in HTML, the language needed a way 
of informing the browser how to read the document structure. For example in a set of hospital records, the browser 
would need to know when a patient record ended and a new record started. Further the browser would need to know 
what type of data defined a patient, such as name, physician’s name, room number or the medication they were 
taking. XML solves this problem through the use of the Document Type Definitions (DTD). The DTD (see Figure 
1) defines not only the overall structure of the document, but also individual elements within the document 

<?xml encoding=‘TJS-ASCn”?> 

<!ELEMENT patientRecords (patient)*> 

<!ELEMENT patient (name,doctor,room)> 

<!ELEMENT name (#PCDATA)> 

<!ELEMENT doctor (#PCDATA)> 

<!ELEMENT room (#PCDATA)> 

Figure 1: Sanq)le Document Type Definition 

In this example the DTD defines the document's data as a collection of patientRecords. These patient 
records are a compilation of individual patients that are identified with a name, a doctor and a room number. The 
DTD then further refines that definition in that all of those elements are typed as PCDATA (parsed character data). 

A browser would have no problem using this data to correctly display the information . Further, we could easily 
determine the number of b^ currently occupied, patient names, and which physicians were caring for them. 

The DTD can be carried by the document so that the browser could properly determine the unique structure 
of the document. However, the DTD can also be referred to as an external file locat^ on the Internet, which could 
define a whole class of documents. As long as the documents are properly formatted, a browser could determine 
how to use them. Consequently, XML is perfectly designed to represent an e. e. cummings poem, as well as, a 
student record database. Using open standards, schools would be able to maximize their installed technology base 
while integrating effective and congruent instructional solutions. 

I>evelopmg the prototype 

XML provides a good model for courseware development because it allows content e?q)erts, software 
developers and even classroom teachers access to the same educational material. Because of the DTD, XML 
provides a mechanism for shaping and disseminating the required protocols to standardize the delivery of 
instructiona] content 

In order to explore the implications of how the new open protocol approach to authoring would work we 
selected a simple multiple-choice exam to prototype and study. This type of exam is used in a variety of educationa] 
settings and is generally well understood. Multiple-choice exams are constructed around a statement with a set of 
possible answers of vdiich only one an^er is correct. 

We configured a Pentium 166 to run the Apache Web Server 1.3 under Red Hat Linux 6. 
The Pentium was also configured to use the Blackdown Java port of Version 1.1.7 for Linux. A 
simple threaded Java Server was authored that processed individual student and exam data and 
saved it to a file located on the server. In addition, we used the IBM XML Classes 1.1.16 to 
facilitate the parsing of the exam data within the Java Applets. The browser for all the groups in 
the study was Microsoft’s Internet Explorer 4.5 for the Macintosh. Internet Explorer allows for 
local access of additional class resources through the Java Preference window. 
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The Study 



XML and Java are new technologies. We were initially interested in investigating the effect of bandwidth 
on user performance. Secondly we were interested in some basic screen layout issues. We wanted to investigate the 
effect of item layout on student performance in a multiple-choice test. 

The study was designed as a usability study that investigated the issues of screen layout 
and bandwidth on student performance. Usability testing (Horn, 1999) uses a mixed mode methodology that 
incorporates both quantitative as well as qualitative methods. Therefore, besides server log data we also video taped 
each exam session, maintained a session journal and interview students at random. 

Th® first group used a Java Applet that accessed local XML classes through the 
iu^^dual student's browser that used a non-wrapping screen layout as illustrated in Fieure 2. 



second group as illustrated in Figure 3, also accessed the set of local classes through the browser. The 
difference in this group was in the screen layout, which used a centered wrapping text design. 

Groups 3 and 4 accessed the reqtiired XML classes that were embedded within the Java 
Applet code. This resulted in an applet that was 17.55 times larger for groups 3 and 4 in contrast 
to groups 1 and 2 (35 1 IMobytes versus 20 Kilobytes). Group 3 used the non-wrapping text 
interface while group 4 used the centered wrapping text interface. 

Partacipants: The participants for this study were 68 pre-service teachers taking a technology integration course. 
During the course the students are required to pass a simple multiple-choice test based on p rimari ly factual 
mformation from the textbook, class lectures, discussions and software specific skills. 

^teriab First we created the Document Type Description for the exam as described in Figure 4. XML allows for 
the definition of new data structures. 



<!ELEMENT exam (question)*> 

<!ELEMENT question (query ,items,answer)> 
<!ELEMENT query (#PCDATA)> 
<!ELEMENT items (a,b,c,d)> 

<!ELEMENT a (#PCDATA)> 

<!ELEMENT b (#PCDATA)> 

<!ELEMENT c (#PCDATA)> 
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Figure 2: Non-wrapping items 



<!ELEMENT d (#PCDATA)> 




<!ELEMENT answer (#PCDATA)> 

Figure 3: Document Type Definition (DTD) 
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Figure 4: Wrapping item list 



The data set was then encoded using the DTD as the fiamework and saved in a simple text file 
on the server. We designed a test bank of 25 questions that was based on factual information 
taken firom the textbook, class lectures and discussions. For example, consider the following 
question: 

<question> 

<query>One kilobyte is equal to bytes of data,</query> 

<items> 

<a>1000<;/a> 

<b>1024</b> 

<c>2000</c> 

<d>2224</d> 

</items> 

<answer>a</answer> 

</question> 

Notice how the question is designed. XML requires complete start and end tags, so that each question is 
required to be marked by <question> and the </question> tag in order to create a weD-fonned document Remember 
that the exam is made vp of questions and each question has a query, item and answer elements. As we can see the 
query is a simple string that makes iq) the actual question. Next we define the four herns from which the student 
must select the correct item. The answer is provided in the answer tag. This allows the ^iplet to track the individual 
student performance within the exam. 

Procedure 

There were 24 sequentially numbered Macintoshes in the computer lab. As a student entered the lab they 
were asked to sit at the next open computer in sequence starting with computer 0 1 . The instructor proctored the 
exam and instructed them to open and read a simple web page located locally on their machine. The web page 
consisted of a set of instructions and acted as a gateway so that they would enter one of the four groups. Half of the 
machines accessed either the remote or local classes and half used the non-wr^)ping or wrapping screen layout 
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Once the student had read the instructions they continued to the next screen that 
displayed a simple demographics survey as shown in Figure 5. The students entered their name, 
the computes number, their sectio n, age, gender, and es timated int ernet usage. 
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Once the student had finished answering the survey questions, fiiey began the exam. 

Each student was presented with 25 questions in a random sequence. Once they had 
answered the questions they were presented with their score and then instructed to log off The 
data was then transmitted to the server for recording. In addition to the demographic data we 
recorded the time that the demographic page was displayed, the order of questions, the student's 
individual responses and the student's total exam score. By identifying each individual compute 
number we were able to correlate the access time based on the server logs for each student. 

These were then arrayed with the other data. In addition to the server-based data we also 
videotaped each exam session, logged any problems that the students may have had. At the 
conclusion of the exam session we also interviewed a group of randomly selected students. 

Results 

Results indicate that an open systems qiproach is a feasible way to design instructional tools. Data 
encapsulation using XML was easily created, m a intaine d and modified thereby allowing for a seamless interaction 
on the part of fiie faculty. The students reported on the exit interviews that taking the exam was easy and they liked 
the convenience of immediate scores. 

One techmcal problem did surface while using the software. The current version was 
designed so that the students could not return to a previous question once they had posted their 
selection. This was an option that students consistently voiced in the exit interview. However, 
only one student attempted to return to a previous question by clicking on the browser's back 
button. To her dismay the exam restarted. That was a design flaw in the implementation. One 
possible solution woidd be put the exam applet in a separate window and disable the back button. 

The location of the required classes was not a factor in the student exam performance. 

On average the Java Applet took about 5.5 minutes to load once the students accessed the web 
site. Students who used the remote classes scored, on average, 21.8 while those that used the 
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local classes had an average score of 22.0. This difference was not statistically significant (p — 
.81). 

According to the data, XML is suitable for developers of instructional software because 
we found no performance difference in using local or remote classes. If you can control the lab 
situation local classes would be preferable simply to reduce server load whereas if you are 
delivering the exam over the internet then you can guarantee access by including the required 
classes. Security still poses a problem, however. The Apache web server supports session 
tracking so that we can program a more robust authentication scheme for a more secure access. 

Educational Implications 

The implications for an open protocol-based educational technology system are cruci^. First it separates 
content from the delivery technology which allows die development of appropriate educational matenal regardless 
of the available delivery technology. Developers would be free to concentrate on effective and ^>propriate 
instructional materials targeted to what teachers need for die classroom. Further it would provide teachers with the 
ability to construct, modify or share computer-based materials for dieir classrooms. This would reduce the time 
needed to create course materials thereby substantially lowering costs. Much work needs to be done, because the 
Internet was developed to siqjport science and industry. Issues of message structure, educational protocols and 
effective content handlers must be developed before die real transformation takes hold. These items keyed the rapid 
rise of the Internet for business and science because diey created portable sources of mformation. The tool you 
choose to complete a job is intimately tied to your success or failure. Matching user skill with the correct tool 
creates an optimal situation guaranteeing die most effective, efficient and elegant solution. (Simon, 1996). If 
education is to emulate die growth of die Internet, diere is a need for similar tools that truly match the needs of 
instruction. 
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